Skip to content

Address SPDD safe-output spec gaps - #55441

Merged
pelikhan merged 11 commits into
mainfrom
copilot/spdd-daily-spec-work-plan-2026-08-24
Aug 24, 2026
Merged

Address SPDD safe-output spec gaps#55441
pelikhan merged 11 commits into
mainfrom
copilot/spdd-daily-spec-work-plan-2026-08-24

Conversation

Copilot AI commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Daily SPDD review found stale security-architecture implementation references, missing scratchpad-removal tracking, weak replace_label normative language, and missing coverage for a label-policy race before setLabels.

  • replace_label safeguard
    • Re-checks label_to_add against allowlist/blocklist policy immediately before the GitHub setLabels write.
    • Rejects blocked or no-longer-allowed labels without calling the write API.
const preWriteAddValidation = validateSingleLabel(labelToAdd, configAllowedAdd, blockedPatterns, "label_to_add");
if (!preWriteAddValidation.valid) {
  return { success: false, error: preWriteAddValidation.error };
}
  • Coverage

    • Adds JS handler coverage proving a mid-flight blocklist change prevents setLabels.
    • Adds Go formal-model coverage for the same no-write invariant.
  • Spec sync

    • Replaces stale pkg/workflow file references with current split-out implementation files.
    • Adds matching last-validated markers to the security architecture summary and validation docs.
    • Strengthens replace_label RFC-2119 language and cross-links shared API failure semantics with outcome evaluation.
    • Adds SPDD-owned scratchpad removal tracking for the 2026-09-21 deadline.

Generated by 👨‍🍳 PR Sous Chef · pi · gpt54 · 5.92 AIC · ⌖ 7.97 AIC · ⊞ 8.7K ·
Comment /souschef to run again

Copilot AI linked an issue Aug 24, 2026 that may be closed by this pull request
10 tasks
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix stale implementation-file references in security architecture spec summary Address SPDD safe-output spec gaps Aug 24, 2026
Copilot AI requested a review from pelikhan August 24, 2026 16:25
@pelikhan
pelikhan marked this pull request as ready for review August 24, 2026 17:23
Copilot AI balanced review requested due to automatic review settings August 24, 2026 17:23
@github-actions

github-actions Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions

github-actions Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Ponytail Reviewer failed. Please review the logs for details.

Warning

Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding.

What happened

The threat detection engine failed to produce results.

Review the workflow run logs for details.

Generated by Ponytail Reviewer for #55441

@github-actions

github-actions Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Design Decision Gate 🏗️ completed the design decision gate check.

No ADR enforcement needed: PR #55441 does not have the 'implementation' label and has only 27 new lines of code in business logic directories (threshold: 100).

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions

github-actions Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel skipped because pre-fetch PR data was unavailable: unable to fetch test file diff

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

Copy link
Copy Markdown
Contributor

Comment Memory

reviewed_at: 2026-08-24T17:27:00Z
review_event: COMMENT
top_themes:
  - no actionable blocking issues found in changed lines
  - replace_label pre-write revalidation is covered in JS and formal tests
  - spec sync updates are documentation-only and internally consistent
files_reviewed:
  - actions/setup/js/replace_label.cjs
  - actions/setup/js/replace_label.test.cjs
  - pkg/workflow/replace_label_formal_test.go
  - specs/replace-label-spec.md
  - specs/safe-output-outcome-evaluation.md
  - specs/safe-outputs-scratchpad-removal.md
  - specs/security-architecture-spec-summary.md
  - specs/security-architecture-spec-validation.md
comment_count: 0

Note

This comment is managed by comment memory.

It stores persistent context for this thread in the code block at the top of this comment.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔎 Code quality review by PR Code Quality Reviewer · pi · gpt54 · 4.84 AIC · ⌖ 6.79 AIC · ⊞ 7K ·
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict

Non-blocking: the changed lines tighten replace_label policy enforcement and keep the JS/tests/spec updates aligned.

Highlights
  • I checked the new pre-write label_to_add validation in actions/setup/js/replace_label.cjs; it closes the mid-flight blocklist/allowlist race without introducing an extra write path.
  • The added JS and Go formal tests cover the new no-write invariant well enough for the behavior changed here.
  • The remaining spec edits are documentation sync and wording improvements; I did not find a changed-line contradiction that would block merge.

🔎 Code quality review by PR Code Quality Reviewer · pi · gpt54 · 4.84 AIC · ⌖ 6.79 AIC · ⊞ 7K
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Impeccable Review — harden + audit

The PR closes SPDD safe-output spec gaps by adding a pre-write re-validation check in the replace_label handler and mirroring the invariant in formal Go tests and spec documents. Changes are well-targeted and the test coverage correctly exercises the TOCTOU scenario.

Non-blocking observations

JS test relies on shared mutable array reference — the test passes a bare [] as blocked to both the handler config and the mutation hook. The test works because blocked.push("done") mutates the same array the closure captured. This is correct today but subtle and fragile: if the handler ever copies the blocklist at init time the test would silently pass for the wrong reason (no error thrown, setLabelsCalls would be 1).

Suggestion (non-blocking)

Consider asserting blocked.length > 0 after the handler call to document the expected mutation state, or modeling the mid-flight change via a callback function rather than a shared mutable array so the test's intent is self-documenting.

Spec sync note placement — the new ### 8.7 Sync Notes section in replace-label-spec.md appears after the staged-mode content (§8.6). A reader following top-to-bottom reaches it after the --- break. Consider whether it belongs alongside the REST failure section (§7) for better discoverability, though this is purely organizational.

No blocking issues. The core invariant (re-validate before write, reject without calling the write API on blocklist hit) is correctly implemented and tested in both JS and Go layers.

🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · copilot · sonnet46 · 27.2 AIC · ⌖ 9.07 AIC · ⊞ 6.2K

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Strengthens replace_label policy enforcement, coverage, and safe-output specification synchronization.

Changes:

  • Revalidates labels before writes and adds regression coverage.
  • Updates security architecture references and validation markers.
  • Strengthens normative language, cross-links, and scratchpad-removal tracking.
Show a summary per file
File Description
actions/setup/js/replace_label.cjs Adds pre-write label validation.
actions/setup/js/replace_label.test.cjs Tests mid-flight blocklist changes.
pkg/workflow/replace_label_formal_test.go Models the no-write invariant.
specs/replace-label-spec.md Adds normative safeguards and sync guidance.
specs/safe-output-outcome-evaluation.md Strengthens API-failure requirements.
specs/safe-outputs-scratchpad-removal.md Assigns removal tracking ownership.
specs/security-architecture-spec-summary.md Updates implementation references and validation marker.
specs/security-architecture-spec-validation.md Adds the matching validation marker.

Review details

  • Files reviewed: 8/8 changed files
  • Comments generated: 4
  • Review effort level: Balanced

Comment thread actions/setup/js/replace_label.cjs Outdated
}
const newLabelNames = [...new Set([...currentLabelNames.filter(n => n !== labelToRemove), labelToAdd])];

const preWriteAddValidation = validateSingleLabel(labelToAdd, configAllowedAdd, blockedPatterns, "label_to_add");

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a4778c5 and follow-up commits: replace_label now fetches pre-write state, revalidates both labels against live policy, recomputes the label payload, and then calls setLabels inside each retry attempt. Policy rejections are explicitly non-retryable.

Comment thread pkg/workflow/replace_label_formal_test.go Outdated

### 8.7 Sync Notes

The REST failure and retry semantics in [Section 7](#7-error-handling) are mirrored by the `replace_label` outcome-evaluation rules in [`safe-output-outcome-evaluation.md` Section 30](safe-output-outcome-evaluation.md#30-replace_label). Changes to `404`, `5xx`, or `429` handling in either document SHOULD be reviewed against the other document in the same change.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6b88f1c: specs/replace-label-spec.md now defines explicit 404 and 5xx/timeout/transport handling in Section 7.

3. If the API returns rate-limit responses (`403` exhaustion or `429`), outcome evaluation workers **MUST** classify as `pending` and reschedule evaluation using the reset window.
4. While any transient API failure condition exists, outcome evaluation workers **MUST NOT** emit `accepted` or `rejected` for label replacement state.

**Sync note:** Keep the API failure safeguards above aligned with [`replace-label-spec.md` Section 7](replace-label-spec.md#7-error-handling), which defines the shared `404`, `5xx`, and `429` REST failure semantics for `replace_label`.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6b88f1c: the cross-reference is now backed by explicit 404 and 5xx/timeout/transport rules in replace-label-spec.md Section 7.

@github-actions github-actions Bot mentioned this pull request Aug 24, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /tdd and /codebase-design — requesting changes on correctness issues in the new tests.

📋 Key Themes & Highlights

Key Themes

  • Test semantics (×2): Both the JS and Go mid-flight-blocklist tests have subtle issues that mean they pass for reasons other than what they claim to test (JS reference mutation; Go append returning a new slice). These don't cause false negatives today, but they reduce regression confidence.
  • Implementation placement vs spec wording: The pre-write guard fires before fetchIssueState + beforeState reads, not immediately before setLabels. This diverges from RL-049a's "immediately before invoking PUT" requirement.
  • Helper signature breadth: formalRunReplaceLabel grew from 2 to 6 positional parameters; an options struct would keep it navigable.

Positive Highlights

  • ✅ RL-049a is a well-scoped, normative addition that closes a real race window
  • ✅ Cross-linking replace-label-spec.mdsafe-output-outcome-evaluation.md for shared error semantics is good long-term hygiene
  • ✅ Replacing stale safe_inputs_parser.go/safe_outputs_config.go references with the current split-out files is a clean, accurate fix
  • ✅ Rate-limit (429) classification as pending with reset-window rescheduling is a meaningful improvement to the outcome-evaluation table

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet46 · 43.1 AIC · ⌖ 10.2 AIC · ⊞ 7.6K
Comment /matt to run again

const result = await handler({ label_to_remove: "in-progress", label_to_add: "done" }, {});

expect(result.success).toBe(false);
expect(result.error).toContain("blocked pattern");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] The "mid-flight blocklist change" test simulates mutation via a shared JS array reference, not an actual config-reload path. This means the test passes for the wrong reason: it's testing JS reference semantics rather than proving a real re-validation occurs before the write.

💡 Suggestion

The test works because blocked is captured by reference in the main() closure and is mutated before validateSingleLabel runs in the pre-write check. But this only holds if the implementation never copies blocked before that check.

A more robust test would set blocked: ["done"] at config-load time with no mutation, proving the pre-write guard fires regardless of when the config was established:

it("should reject label_to_add before setLabels when it is blocked", async () => {
  let setLabelsCalls = 0;
  mockGithub.rest.issues.setLabels = async () => { setLabelsCalls++; return { data: [] }; };

  const handler = await main({ allowed_add: ["done"], blocked: ["done"] });
  const result = await handler({ label_to_remove: "in-progress", label_to_add: "done" }, {});

  expect(result.success).toBe(false);
  expect(result.error).toContain("blocked pattern");
  expect(setLabelsCalls).toBe(0);
});

If the intent truly is to test mid-flight config reload, the handler would need to re-read config from an external source at write time — which the current implementation does not do.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a47dd6e: the JS regression now models a config reload by replacing config.blocked during the second GET, and the handler re-reads current policy at the pre-write gate.

// (issues.setLabels). In staged mode the handler must return before reaching
// onWrite; this is the invariant asserted by TestFormalStagedMode_NoWriteAPI.
func formalRunReplaceLabel(staged bool, onWrite func()) formalReplaceLabelOutcome {
func formalRunReplaceLabel(staged bool, labelToAdd string, allowedAdd, blocked []string, beforeWrite func() []string, onWrite func()) formalReplaceLabelOutcome {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/codebase-design] formalRunReplaceLabel now has 6 parameters including a beforeWrite func() []string callback — a significant complexity increase for a helper that previously had 2. This makes the formal-model tests harder to read at a glance.

💡 Suggestion

Consider introducing a struct or options type so call sites are self-documenting:

type formalRunOpts struct {
    LabelToAdd  string
    AllowedAdd  []string
    Blocked     []string
    BeforeWrite func() []string
    OnWrite     func()
}

func formalRunReplaceLabel(staged bool, opts formalRunOpts) formalReplaceLabelOutcome { ... }

Existing call sites that don't need BeforeWrite can omit the field (zero value is nil). This keeps the interface shallow as new test scenarios are added.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6b88f1c: formalRunReplaceLabel now takes a formalRunReplaceLabelOpts struct so call sites are self-documenting.

false,
"done",
[]string{"done"},
blocked,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] TestFormalBlockedLabelAddedViaReplaceLabelMidAllowlistChange_NoWriteAPI initialises blocked := []string{} and then passes append(blocked, "done") from beforeWrite — but append on an empty slice returns a new slice, so the outer blocked variable is still empty after beforeWrite returns. The formal model correctly rejects the label because formalRunReplaceLabel reassigns its local blocked parameter, but the test name implies mid-flight mutation of the original slice, which isn't what happens.

💡 Suggestion

Either make the intent explicit:

// beforeWrite returns a new blocked list — simulates a policy reload
func() []string { return []string{"done"} },

Or, if the intent is to prove the pre-write check fires even when blocked was empty at entry, rename the test and simplify beforeWrite to make that invariant clear. Misleading variable setup reduces confidence that the test would catch a real regression.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6b88f1c: the formal test is renamed for the mid-blocklist scenario and BeforeWrite now returns a new blocked list to model policy reload explicitly.

Comment thread actions/setup/js/replace_label.cjs Outdated

const preWriteAddValidation = validateSingleLabel(labelToAdd, configAllowedAdd, blockedPatterns, "label_to_add");
if (!preWriteAddValidation.valid) {
core.warning(`label_to_add validation failed before setLabels: ${preWriteAddValidation.error}`);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/codebase-design] RL-049a says re-check MUST happen "immediately before invoking PUT .../labels" but the implementation inserts the check after newLabelNames is computed (line ~204) and before core.info + fetchIssueState — which itself makes additional read API calls. Those reads mean the check is not the last operation before the write.

💡 Suggestion

Move the preWriteAddValidation block to immediately before the setLabels call rather than before fetchIssueState. This ensures the spec wording "immediately before invoking PUT" is satisfied by the code structure, not just by coincidence of ordering. A comment marking the location as the spec-required gate would also help future reviewers.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed across a4778c5 and follow-up commits: the final policy gate now runs inside each retry callback after the fresh state read and immediately before setLabels.

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot The Ponytail Reviewer workflow failed for this PR.

Forward progress needed

  • Please inspect the failing check: agent
  • If the failure is transient or tooling-related, rerun or adjust the PR so the check can complete successfully.
  • After addressing the failure, run the pr-finisher skill to move the PR toward merge readiness.

Generated by 👨‍🍳 PR Sous Chef · pi · gpt54 · 5.92 AIC · ⌖ 7.97 AIC · ⊞ 8.7K ·
Comment /souschef to run again

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Copilot AI and others added 7 commits August 24, 2026 18:06
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot AI commented Aug 24, 2026

Copy link
Copy Markdown
Contributor Author

@copilot The Ponytail Reviewer workflow failed for this PR.

Forward progress needed...

Inspected the failing Ponytail Reviewer job logs. The failure was infra/tooling-related (OpenAI proxy not configured) and emitted no review items; I ran pr-finisher, addressed the actionable review threads, pushed fixes, and validated locally.

Copilot AI requested a review from gh-aw-bot August 24, 2026 18:24
@pelikhan
pelikhan merged commit 55d6bae into main Aug 24, 2026
5 checks passed
@pelikhan
pelikhan deleted the copilot/spdd-daily-spec-work-plan-2026-08-24 branch August 24, 2026 18:37
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v0.87.5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[spdd] Daily spec work plan - 2026-08-24

4 participants